Control of social robot based on prior character portrayal

ABSTRACT

A method and apparatus for controlling a social robot includes providing a set of quantitative personality trait values, also called a “personality profile” to a decision engine of the social robot. The personality profile is derived from a character portrayal in a fictional work, dramatic performance, or by a real-life person (any one of these sometime referred to herein as a “source character”). The decision engine controls social responses of the social robot to environmental stimuli, based in part on the set of personality trait values. The social robot thereby behaves in a manner consistent with the personality profile for the profiled source character.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 16/258,492, filed Jan. 25, 2019, which is a continuation ofInternational (PCT) Application Serial No. PCT/US2017/044038 filed onJul. 26, 2017, which claims priority to U.S. Provisional ApplicationSer. No. 62/467,709 filed Mar. 6, 2017 and to U.S. ProvisionalApplication Ser. No. 62/367,335 filed Jul. 27, 2016, each of which areincorporated herein by reference in their entireties for all purposes.

FIELD

The present disclosure pertains to social robots, and to methods forcontrolling social robots.

BACKGROUND

As used herein, a “social robot” is an interactive machine designed toevoke an emotional response from a human or other social animal thatinteracts with it, by exhibiting behaviors that are recognizably socialin nature for the humans or other social animals with which the socialrobot is designed to interact. A social robot may be implemented as areal machine with the power to move at least part of its robotic body ina real environment. In addition, a social robot may be implemented as avirtual machine that can be expressed only in a virtual environment,such as, for example, as a robotic character in a video gameenvironment, in an immersive virtual reality environment, or in anaugmented reality environment. As used herein, the term “social robot”generally encompasses both real and virtual social robots. Whenspecifically pointing out one of these classes of social robots, theterms “real social robot” or “virtual social robot” will be used herein.

Social robots can exhibit personality. As used herein, “personality”means an entity's patterns of behavior that are perceivable by othersand relevant to social interaction, such as are distinctlycharacteristic of the entity. Different entities may share the samepersonality, by exhibiting the same patterns of behavior in similarcircumstances. Social robots, for example, often share the same orsimilar programming, and therefore exhibit the same or similarpersonalities. Furthermore, the personalities of social robots may tendto be predictable or less interesting than human personalities.

The entertainment industry is expert at development of personalities forcharacters, and at expression of those personalities under diversecircumstances. Entertainers compete for audience attention by presentingcharacters with captivating personalities, and stories that exercisethose characters in interesting circumstances. The characters portrayedare often memorable, whether heroes, villains, clowns or some othercharacter type. In addition, production studios develop and accruenumerous assets that record characters' behaviors and other personalattributes in various forms, e.g. screenplays, scripts, abstracts, storyboards, pre-viz software representations, screen tests, other video orfilm sequences, subtitles, closed captioning with descriptive text, artwork, or other data. However, methods for systematically applying thisexpertise in character development and studio assets to social robotsare not known.

It would be desirable, therefore, to provide methods for endowing socialrobots with more interesting and varied personalities, for example, withpersonalities such as have been expertly portrayed by skilled actors,writers, and other content creators, and found appealing by audiences,and for controlling the operation of social robots to express moredistinctive and interesting personalities.

SUMMARY

This summary and the following detailed description should beinterpreted as complementary parts of an integrated disclosure, whichparts may include redundant subject matter and/or supplemental subjectmatter. An omission in either section does not indicate priority orrelative importance of any element described in the integratedapplication. Differences between the sections may include supplementaldisclosures of alternative embodiments, additional details, oralternative descriptions of identical embodiments using differentterminology, as should be apparent from the respective disclosures.

A method and apparatus for controlling a social robot includes mapping aset of robot behaviors to a set of quantitative personality trait valuesusing a first personality profiling function. In addition, a corpus ofproduced movie, television, literary or game products featuring aparticular character may be processed and mapped onto the same orequivalent set of personality trait values, using a second personalityprofiling function. In alternatives, a fictional character is producedbased on an arbitrarily chosen set of personality trait values, or theset of personality trait values is derived from speech and behavioraldata acquired from a real-life person, using a third personalityprofiling function. However, the set of personality trait values isobtained, it is provided as input to a decision engine of the socialrobot. The decision engine controls social responses of the social robotto environmental stimuli, based in part on the set of personality traitvalues. The social robot thereby behaves in a manner consistent with thepersonality profile for the profiled character.

A social robot comprises a set of modules under common control with adefined relationship to one another, wherein the set as a whole isdesigned to interact socially with a human (or other, e.g. pet dog)companion. For example, a Jibo™ is a passive torso with a swivelinghead, video-screen face, ears (audio input) and static (non-animated)speech output. A smartphone provides an example of another type of robotconsisting of a passive frame, video-screen face, ears (audio input) andstatic (non-animated) speech output. Robots such as Asimo™ are morecomplex, have non-video facial features, and means of locomotion and forarticulating extremities. Modes of expressing a character's personalityvia a smartphone would be different from modes of expressing personalityvia a humanoid robot like Asimo. Regardless of the specific mode ofexpression, a decision engine controls output of the social robot inresponse to input based at least in part on a personality profilederived from a character portrayal in a fictional work, dramaticperformance, or from a real-life person.

The personality profile may include a list or other set of quantitativevalues that are derived by scoring character metadata against a scale ofsome kind. Various different personality scales are known in the art foranalyzing and profiling human personalities. A known scale may be used,or a proprietary scale may be developed that is optimized for control ofthe social robot.

In an aspect, a social robot designer may select a set of personalityattribute values that can logically be mapped to social behaviors of arobot, using a probability function. For example, a set of personalitytrait values may be expressed in a scale between polar opposites. Forfurther example, for a trait between open-minded and conservative, avalue of 70% may mean that, when presented with a choice betweenopen-minded and conservative choices, 70% of the character's choices areopen-minded choices, and 30% are conservative choices. Further examplesof polar opposite traits include: extroverted/introverted, fast/slow,focused/distracted, diligent/lazy, humorous/serious,curious/disinterested, apathetic/caring, vulgar/polite,dominant/submissive, cynical/naive, honest/dishonest, open/secretive,loud/soft-spoken, superficial/profound, wise/foolish,adventurous/cautious, stubborn/compliant, or foolish/prudent.

In an aspect, a personality profile may include other factors forvarying the set of quantitative personality values based oncharacteristic moods. For example, a character may tend to be grumpy foran hour after waking up, may be giddy in the presence of a happy woman,happy and well-behaved around children, relaxed after 8 pm, and “normal”the rest of the time. All of these different moods may be reflected bydifferent sets of personality trait values. A mood engine determines theoperative set of personality factors at any given point of time, basedon its own set of control factors, examples of which have just beendescribed. Accordingly, a social robot may be made to express moods thatare characteristic of the profiled character.

In another aspect, a robot memory holds data for generating verbalresponses based on a set of characteristic phrases for the characterstored in the memory. Accordingly, the social robot uses phrasing thatis characteristic of the character(s) that the social robot isemulating. In another aspect, the memory further holds data forgenerating motions or configurations of the at least twohuman-recognizable expressive features, based on a set of characteristicmotions or configurations for the character stored in the memory.Accordingly, the manner in which the character moves or configures itsexpressive features will evoke the character that is portrayed.

In another aspect, a social robot memory holds distinct sets ofquantitative personality trait values, each representing a distinctpersonality. The robot may take on different personalities dependingupon its mood. The robot's mood can be determined by one or moreenvironmental triggers, which may be quite varied. For example, arobot's mood could cause it to behave differently depending upon withwhom it is at the moment being social. So, considering Myers Briggssorts of personalities and behaviors, if the same robot were conversingor otherwise behaving with a controlling extrovert, someone whointerrupts and finishes others' sentences, then the robot couldadaptively behave in a congruent way. If the robot were insteadconversing or otherwise communicating with a mild mannered, soft-spokenintrovert who does not usually speak unless spoken to, and who prefersquestions to directives, then the same robot could adaptively behave ina congruent way that fits those circumstances. For further example, therobot may adapt different linguistic conventions and behaviors that maybe more or less appropriate depending on the age(s) of the interactinghuman(s), such as an adult or child. When interacting with groups, therobot's behavior might be varied according to the dynamics of theconversations, i.e. who is speaking and who is answering or otherwiseprimarily engaged. This variance may be controlled by a mooddetermination engine that switches the controlling robotic personalityin response to any desired mood trigger or set of triggers.

In another aspect, an appropriately configured personality profileenables a social robot to take on the personality profile of characterspreviously portrayed in cinema, video games, video serials, novels orgraphic novels, or any other form of expression that portrays characterswith discernible personalities. In addition to personality, suchcharacters may possess distinctive characteristics such as appearance,dress or costume, grooming or hygiene habits, hair or make-up styles,type or form of accessories worn or carried, physical mannerisms,gestures, stances or poses, idiosyncrasies, behavioral quirks, likes anddislikes, facial expressions, catch phrases, and speech characteristics.Speech characteristics may include, for example, inflection, volume,cadence, pitch, timbre and tonal qualities of the characters' voice.Social robots may mimic some or all of these characteristics, dependingon the social robot capabilities and the desired effect. The style ofpersonality mimicry may vary. For example, a social robot might beprovided with a personality profile causing it to behave ‘in the styleof’ or otherwise obviously influenced by a character without behavingexactly like the character in question. The words need not be quotes;the gestures and mannerisms need not be exact imitations. In fact,mild-to-gross exaggeration can be an effective and entertaining way ofbehaving ‘in the style of’ or otherwise being obviously influenced by,so that no one would say that the behavior is exactly like the characterin question, but such behavior still surely reminds us of thatIntellectual Property (IP) character, nevertheless. For further example,characteristic traits may be combined in different ways to producecomposite effects, such as, for example, a social robot that acts andtalks like Spiderman while using Darth Vader's voice, exhibitingpersonal or behavior traits of both characters.

Initial sets of quantitative personality trait values—personalityprofiles— may be modified in response to experiences of a social robot.The robot may be configured to learn different trait values or todevelop its own personality through machine learning or another AI. Forexample, a social robot may identify and add to memory newcharacteristic phrases, mannerisms, gestures, motions or other behaviorson-the-fly. The repertoire of personality profiles may be contouredadditively by identifying, deriving, and then using salient intonationcontours, phrases, or cadences of spoken language interactions with thehuman(s) at issue. Additionally, a robot's behavior may include‘vocalizing’ that is not actually a language but that serves somecommunication purpose. The vocalizing is not in an actual living‘language’ but either sounds like it serves some communicative purposeor is in a fictional or dead language that obeys certain rules. Eitherway, the robot and a human with which it interacts obey certainconversational conventions, such as waiting for each other to finish‘speaking’ before going on.

Configurations of human-recognizable expressive features, in contrast tospoken language, may also be learned by the robot through machinelearning or other Al by identifying, deriving, and then adding to memorynew characteristic mannerisms or other relevant behaviors on-the-fly sothat the repertoire of personality profiles could be contouredadditively through interactions with the human(s) at issue, and, again,all these could be used in novel utterances or behaviors ‘in the styleof’ or otherwise influenced by the IP character(s) at issue, for exampleby using a stochastic process such as, for example, a Markov chain orvariation thereon. Personality trait values may provide probabilitiesfor outcomes of a stochastic process for that purpose.

In some embodiments, a robot may use a ‘test’ program to identifyinteracting human or environments in which interaction is taking place.For example, a social robot might interact with humans using a testprogram. The test program need not take the apparent form of a test.Instead, the test program may be designed to be perceived as a one-timeset-up process, a game, or some other entertaining process that samplesan interacting person's personality, while also entertaining them. Forexample, the test may sample the interacting human's voice, traits,mood, expressions or mannerisms, and store identifying characteristicsin the person's profile for future reference. An initial interactionsetup program might be called into play only initially when interactingwith a human, or perhaps every time a session with the robot commences,so that the robot could determine the interacting human's age, sex,height and build, mobility (i.e. whether the human is able to movefreely, or bedridden, or feeble, or . . . ), spoken language (since therobot could be conversant in several), or other factors. The robot maystore the person's profile, and thereby later recognize that person,e.g. by employing facial recognition software, when that person is againinteracting with the robot. Using voice recognition software, once aperson has been identified, then even if her/his appearance changes soas to render the person unrecognizable visually, the robot may stillmake the identification based on voice.

Similarly, via object recognition or image analysis software, the robotmay determine characteristics of the space which the robot and humanoccupy (e.g. indoors, outdoors, in a small area or a large one, whattime it is, whether it's daylight or nighttime, etc.) so thatappropriate conversation and related behaviors could be best managed.For example, there's probably no need to speak of how lovely the starsappear to be if it's broad daylight, and there's no need to say “Let'sgo for a walk” if the human can't move or is preparing for bedtime. Inthis manner, the robot may either mimic or counterpoint the voice,traits, mood(s), expressions or mannerisms depending upon what iscircumstantially appropriate. If, for example, the robot later detectsat a subsequent meeting for example by identifying through machinelearning or Al or consultation with templates or consultation withdatabases of examples) that the human is depressed, then it could setabout trying to lift the human's spirits by telling a joke, offering toplay a game, playing a favorite song, or other mood-lifting interaction.All of the above interactions may take place virtually, i.e. with therobot and the human(s) in different places, interacting by way of someaudiovisual system—e.g. via Skype.

In another aspect, operation of a social robot is integrated with alocal or remote database of content that is selectively shared with theuser of the social robot, in response to a current stimulus oranticipated future condition. For example, the database of content mayinclude video clips, audio-video clips, or audio clips that are indexedbased on their semantic content. The social robot may be configured tooutput a selected one of the clips for any social purpose. For example,if the social robot senses that its user is unhappy, it may select ahumorous clip from clips of the user's known favorite content orcharacters. For further example, if the social robot is commenting on asensed social situation, for example, to encourage the user to get someexercise, it might play a clip that is marked as appropriate for userswith personal or demographic characteristics that matches the users'that in some way reminds the user to get up and do some exercise.Accordingly, user's may be entertained by the context and selection ofclips selected by the social robot's content selection algorithm. Inmore detailed aspects, and for further example, the robot may:

-   -   learn what type of relevant content clips the current user or        users are likely to pay attention to, for example by identifying        areas, genres, works (prose, poetry, pictures, videos, art,        etc.) that interest the target user(s) by questioning or        otherwise ‘testing’ the target users(s) for interest levels, and        then audio-visually output the selected content clips;    -   wirelessly send selected content clips (or have them sent from a        remote server) to audiovisual display gear (e.g., speakers, a        phone, a tablet, a PC, a smart TV, etc.) independent of the        robot so that the target user(s) can see and/or hear the content        clip synchronized to a social interaction involving the social        robot, or in response to an authorized user request;    -   command a wireless device such as, for example, the Amazon™        Echo™ and Fire Stick™, to tune in or otherwise obtain relevant        content for the target user(s), wherein the content is selected        by the social robot based on user/situational parameters or        selected by the user.

The method and apparatus of the present disclosure may be embodied insocial robots of various types, and in computer networks used to supportoperation of social robots. For example, personality analysis forprofile development may be performed partly or entirely using cloudcomputing resources, with results shared with a local robot for uselocally. The technical description that follows enables all of thefunctional abilities of a social robot that are summarized above.

BRIEF DESCRIPTION OF THE DRAWINGS

The features, nature, and advantages of the present disclosure willbecome more apparent from the detailed description set forth below whentaken in conjunction with the drawings in which like referencecharacters identify like elements correspondingly throughout thespecification and drawings.

FIG. 1A is a block diagram illustrating aspects of a system and methodfor control of a social robot based on a prior character portrayal infiction or performance.

FIG. 1B is a block diagram illustrating additional aspects of a systemand method as shown in FIG. 1A.

FIG. 1C is a block diagram illustrating additional aspects of a socialrobot component of the system shown in FIG. 1A.

FIG. 2 is diagram showing an example of a quantitative data structurefor a personality profile or score based on bipolar opposite traits.

FIG. 3 is a flowchart showing aspects of a method and system forcharacter analysis for developing quantitative personality profiles andcharacteristic social behaviors, which are in turn useful for control ofa social robot based on a prior character portrayal in fiction orperformance.

FIG. 4 is a Venn diagram illustrating aspects of stimulus-responsepairing in connection with operation of a social robot based on a priorcharacter portrayal in fiction or performance.

FIG. 5 is a flowchart showing aspects of a method and system for mappingbehaviors developed by analysis of a prior character portrayal infiction or performance to social behaviors that can be performed by asocial robot.

FIG. 6 is a flowchart showing aspects of a method and system forcorrelating robot behaviors to quantitative personality profilesdeveloped by analysis of a prior character portrayal in fiction orperformance.

FIGS. 7A and 7B are concept diagrams illustrating aspect of quantifiablestimulus and response parameters, respectively, which paired togethercan be used to define a behavior numerically.

FIG. 8 is a table illustrating aspects of a data structure forrepresenting a stimulus or response numerically.

FIG. 9 is a concept diagram providing a graphical illustration of a 4×6numeric stimulus value paired to a 4×4 numeric response value, whereinthe illustrated stimulus-response pair (a behavior) correlates to asingle 1×8 numeric personality score for the behavior.

FIG. 10 is a flowchart showing aspects of a method and system forcharacterizing a stimulus, in real time for operation of a social robot,wherein the stimulus correlates to one or more responses based on one ormore personality profiles in use for controlling the social robot.

FIG. 11 is a flowchart showing aspects of a method and system forsetting and changing a current personality profile for a social robot,based on a characterized stimulus and on mood-switching behaviorsdefined in a computer memory operatively coupled to the social robot.

FIG. 12 is a flowchart showing aspects of a method and system forselecting a current response based on a current stimulus and personalityprofile for a social robot.

FIG. 13 is a block diagram showing aspects and components of a realsocial robot.

FIG. 14 is an isometric view showing an example of a real social robot.

FIG. 15 is a block diagram showing aspects and components of a virtualsocial robot.

FIG. 16 is combined block diagram and rendering showing an example of avirtual social robot.

FIG. 17 is a concept diagram illustrating allocation of a response tocoordinated real and virtual social robots.

FIG. 18 is a flowchart illustrating aspects of a method for control of asocial robot based on a prior character portrayal in fiction orperformance, based on a quantified personality profile.

FIG. 19 is a block diagram, illustrating an example of a social robotconfigured for performing the method of FIG. 18 .

DETAILED DESCRIPTION

Referring to FIG. 1A, a social robot system 100 for controlling a socialrobot based on prior character portrayal in fiction or performance isillustrated. It should be appreciated that the system 100 may be adaptedfor use with any character portrayal or performance, including that of areal person who is portraying his or her own natural character. Thesystem includes various modules and data structures that may beincorporated in hardware, software, firmware, or in some combination ofthe foregoing. The hardware, software, firmware, or combination may beimplemented locally in the social robot itself, or in part usingcomputer processing power that is in communication with the social robotusing a computer or data communications network.

The system 100 is organized into three subsystems: a character andpersonality profiling subsystem 120, a robotic behavior mappingsubsystem 140, and a real-time social robot behavior module 160. Thecharacter and personality profiling subsystem 120 may be used in advanceof robot operation, to develop one or more personality profiles andassociated distinctive physical character traits, for example, clothingand accessories, appearance, and voice qualities for use during robotoperation. The subsystem 120 may be more briefly referred to as apersonality profiling subsystem. The robotic behavior mapping subsystem140 may also be used in advance of robot operation, to map personalbehaviors and character traits to social robot behaviors where possible,and to correlate all social behaviors of the robot to a personalityscale used for defining the robot's personality profile. The real-timesocial robot behavior module 160 operates in real time when the socialrobot is interacting with its live user or users, causing the socialrobot to express one or more of the personalities profiled by theprofiling module 120 with associated physical character traits that thesocial robot is configured for expressing.

FIG. 1B shows a system and network 180 that instantiates aspects of thesystem 100 for controlling a social robot, using a computer network. Thesystem 180 includes the subsystems for character and personalityprofiling 120, robot behavior mapping 140 and real-time robot behavior,here instantiated in two independently operating robot behaviorsubsystems 160′ and 160″. Each of the robot behavior subsystems 160′ and160″ comprise a virtual agent platform executing, wholly or partially,in processors of their respective, independently operating social robots161′ and 161″. The social robots 161′ and 161″ may be of different typeshaving different capabilities. It should be appreciated that a widevariety of virtual agent platforms may make use of character andpersonality profiles. Each virtual agent platform may receive one ormore character and personality profiles from a data store 188 via aprofile server 190 and Wide Area Network (WAN) 182 or othercommunication network. The data store 188 may be, or may include, acloud storage component. It should be appreciated that the same virtualagent (e.g., social robot 106′ or 160″) may be “skinned” with differentcharacter profiles at different times. In the context of personality orcharacter profiling, to be “skinned” means to act and appear inaccordance with a particular character profile.

Each of the profiles in the data store 188 may correspond to one or morecorresponding character profile stored in data store 186. The characterprofiles in data store 186 may be provided to the behavior mappingsubsystem 140 to develop robot-specific profiles for specific robots ortypes of robots, e.g., social robots 161′ and 161″, held in the datastore 188. The character profiles in data store 188 may each comprisespecifications for a persona, which may include both physical traits(e.g., voice, appearance) and behavioral traits. The specifications aregenerated according to a mathematical model or coding scheme for one ormore virtual agent platforms. The character profiles may be platformagnostic, while being adaptable for use for different virtual agentplatforms. In an alternative, or in addition, character profiles in datastore 188 may be developed for exclusive use with a specific virtualagent platform type. The data store 188 may be instantiated as a single,central, canonical entity in a cloud computing or other network, madeavailable to virtual agent platforms via a generalized applicationprogram interface (API). In addition, or in an alternative, the datastore 188 may be implemented as an integrated or embedded feature ofthird-party platforms, much like an software developers' kit (SDK) withruntime libraries that call for functions as needed,

Each character profile may be developed by the character and personalityprofiling module 120 from raw character source data 184. In an aspect,character profiles may evolve over time in response to new data forcharacter sources, improved or modified character profiling tools, userpreference data, or other factors. Raw character source data 184 mayinclude, for example, video data, scripts, subtitles, stories,screenplays, closed caption information, art work, or other characterinformation stored in a digital, machine-readable form. Machine learningand other processing tools may be used to extract character andpersonality information from digital video or audio data. For example,deep learning (sometimes also called deep machine learning) may be usedto analyze video and audio data to recognize personal identities, bodylanguage, facial expressions, emotional state, object types, languagecontent (e.g., speech recognition) and social context. Analysis mayfurther include processing of natural language to detect higher-levelmeanings such as, for example, sarcasm, irony, or humor. For example, anemotional detection engine such as used by Affectiva™(www.affectiva.com) may be used to detect a source character emotionalstate, as well as an end user emotional state. The emotional state of asource character can be correlated to stimuli as a response, using astimulus-response framework as described herein below. In addition, theemotional state may be treated as a stimulus that is associated with itsown behavioral response. For example, an anger emotional state might becorrelated with swearing or yelling, among other things. From theanalyzed raw information, personal behavior can be understood based onsocial context, nonverbal communication, verbal communication, cause andeffect, or other factors. Character responses can be scored or otherwiserecorded in some tangible, symbolic value using a personality model asdescribed herein, or other suitable description model. When provided toa compatible social robot real-time controller (e.g., modules 160′,160″), the character's personality score or other symbolizedrepresentation of the character's stimulus-response profile causes thesocial robot to behave in a manner that probabilistically emulates thebehavioral personality of the source character, and optionally selectedphysical traits of the source character.

In addition to behavioral personality analysis, analysis of raw data 184may include extraction of physical characteristics that can be mimickedby the target population of social robots. Such mimicry of distinctivephysical characteristics may supply considerable attraction to characteremulation by social robots, being relatively easy for end users torecognize. For example, mimicry of a known character's voice and speechpatterns may be more easily recognized than the same character'sbehavioral tendencies, which may be more subtle or less well-known.Likewise, the source character's appearance, including but not limitedto clothing, accessories, physical form and mannerism may be mimicked bysocial robots. For example, social robots using a video display as abody component (e.g., for the robot face) are capable of closelyreproducing the appearance of the source character's corresponding bodypart. For further example, distinctive body language can be reproducedby articulating robots having an armature similar to the sourcecharacter. Artificial intelligence, including but not limited to deeplearning, may be used to identify any distinctive physicalcharacteristics or the source character, and create a symbolic (e.g.,digital) representation of the physical characteristics that enablessocial robots that are capable of mimicry to mimic those physicalcharacteristics, or some subset of the characteristics, when adoptingthe source character's persona.

The social robots 161′ and 161″ operating the respective behavioralmodules 160′ and 160″ should react in real time to changes in contextand user actions, meaning react in approximately the same amount of timeas a person, so the robot response feels natural. Accordingly, some morecomputationally intensive functions may slow down the robot's responsestoo much, so that the robot seems to lag behind the user input orenvironmental changes. However, peak processing loads may beintermittent, and building robots to handle peak processing demands mayadd unjustified costs. To enable handling of varying processing loadswithout allocating system resources inefficiently, more intensivecomputational tasks may be segregated to a dedicated computational node.For example, supposing that detection of emotional state iscomputationally intensive, detection of the user's emotional state touse as stimulus input for the robot's reaction might be segregated fromother robot processing, and sent to a network node (e.g., to profileserver 190) using a network connection. The processing network node maythen transmit a data signal to the social robot every time a differentemotional state of a user is detected. Such transmissions may act asstimuli for corresponding robot responses, alone or in combination withother input.

Referring to FIG. 1C, any given social robot's (e.g., robot 161′)behavior module 160′ may select a response based on a combination ofdifferent types of stimuli. The present disclosure describes stimulitypes such as events 163 (e.g., user actions, speech, or changes inemotional state) and physical context 165 (e.g. place, time of day,current state of present place). Other stimuli types may include, forexample, a user ID and associated user profile 171, so the robot canexhibit different personality traits to different users, just as peopledo. In addition, stimuli may include remembered social context 167. Forexample, if the robot remembers user preferences, it can apply thisknowledge in positive or negative ways depending on the trait it isexpressing. For further example, the robot may modify its response basedon its level of social intimacy with the person with whom the robot isinteracting or past on specific past experiences, based on rememberedinformation. Accordingly, the robot's characteristic behavior towards auser may change as it collects information from interactions with thatuser.

It should be apparent that real-life social stimuli are complex andmulti-dimensional. Nonetheless, social responses can often be understoodas motivated primarily by a relatively sparse set of stimuli, with mostother stimuli being ignored. Focus on primary motivators is importantfor social robots just as it is for people, to enable appropriate andtimely responses. In social robot systems, focus may be achieved bydesigning the character model based on a limited set of primary stimulithat are most likely to influence social interactions, for example,speech, user identity, user preferences, and time of day. As additionalexperience is gained in robot operation, additional stimuli may be addedto the character model as the model is refined.

Referring again to FIG. 1A, in addition to compiling information aboutdistinctive physical character traits, the personality profilingsubsystem 120 receives behavioral data 124 for a character or person andanalyzes 122 the data 122 based on a predefined or adaptable personalityscale 126. As used herein, “behavioral data” describes or defines setsof stimulus-response pairs, including but not limited to statisticaldata regarding relative frequency of occurrence of any particular pair.Stimuli and responses may be described at any desired level of detail,including different levels of detail. For example, a stimulus andresponse may be generic, such as a stimulus of “being ignored” and aresponse of “seeking to be noticed.” For further example, either or bothof the stimulus may be much more specific, such as a stimulus of hearing“how are you?” paired with a response of “What's it to you!?” Whilebehavior may be characterized in any useful manner, it may be desirableto choose stimuli and responses for profiling a character's personality,that can readily be mapped to stimuli that the social robot is capableof detecting, and responses that the social robot can perform. Inaddition, the behavior should be, or be capable of being correlated to,a social function of the robot.

The source of behavioral data may vary. Useful data should bequantifiable and capable of statistical analysis. Each stimulus shouldbe capable of pairing with more than one response, so that the frequencyof a particular social response can be correlated to a particularpersonality trait. For example, suppose a particular stimulus “A” canresult in any one of responses 1-10. Therefore, if for a certaincharacter a response of “3” occurs 50% of the time, with “7” and “9” 25%of the time each, and the remaining responses 0% of the time, thestatistical tendency can be correlated to a trait, because thecharacter's responses are not random.

Behavioral data may be collected in various ways. A traditional methodis to ask the subject a series of questions involving hypotheticalsocial situations and ask the subject to identify a preferred response.This is not directly possible for fictional characters, but it may bepossible for an actor or other person who is either responsible forcreating the character, or is familiar with detailed expressions of thecharacter, to answer questions on the character's behalf. This manual,approximate approach may be useful for quickly approximatingwell-defined characters. An automated approach to assessment correlatesstatistical patterns in a character's stimulus-response behavior againsta list of personality traits. This requires a robust data set and anautomated analysis engine that is capable of processing the data set,recognizing stimuli and responses, and correctly associating responsesto their proper stimuli, for example, a deep machine learning engine.The robust data set may be provided, for example, by episodes of videocontent, issues of graphic novels or other serial fiction, motionpictures, books, video games with narrative content, or combinations ofthese and similar forms of content. The automated analysis engine may,for example, analyze a script or other narrative writing to assesscharacter personality, using image analysis to recognize facialexpressions and characters, deep learning and neural networks torecognize and correlate stimulus-response pairs, and other artificialintelligence tools. In an alternative, an analyst may manually reviewthe data set, recognize stimulus-response pairs, and score a personalityprofile accordingly.

By whatever method the behavioral data is obtained, the personalityprofiling subsystem 120 evaluates the behavioral data against apersonality scale 126, which may be unique or particular for the robotbeing operated. It should be appreciated that the personality scale may,in effect, prioritize character traits to be emulated by social robots.For example, a personality trait that does not appear in the scale isweighted at zero and is not emulated at all. Traits appearing in thescale may be weighted to emphasize or prioritize selected personalitytraits over others. For example, a humorous-serious trait scale may beweighted more heavily than an agile-clumsy scale, causing the socialrobot to exhibit more humor or soberness (depending on its score forthis trait) and less agility or clumsiness (again, depending on itstrait score) then would be the case if the scales were unweighted.

FIG. 2 shows an example of a data table 200, representing a personalityprofile based on a personality scale 202 (also called a “bipolar scale”)that includes a list of bipolar traits each making a pair of oppositeextremes. Each row (e.g., first five rows shown 230, 232, 234, 236 and238) pertains to a particular pair of opposites traits, representednumerically as a number and its opposite, each pair making one member ofa set of opposite trait pairs. The depicted scale is 8 bits, allowingfor 128 graduations of behavior in the personality profile 200; anydesired number of bits may be used, but more than about 8 bits isunlikely have easily perceived effect on the behavior of the socialrobot. For example, row 230 indicates a 77/128 or 60% tendency for thecharacter to behave as an extrovert, with a 40% tendency towardsintroverted behavior. Row 232 shows a 0% tendency to humor, with a 100%tendency to be serious. Row 234 shows a 39/128 or 30% tendency towardsbeing focused, and a 70% tendency for being bored or interested. Row 236shows a 100% tendency to be dominant, with 0% submissive tendencies. Thelast row 238 shows a 50/50 split of the personality between selfish andunselfish (caring) behavior. Any useful number of traits may similarlybe analyzed and represented in a quantified personality profile table200, or similar data structure.

The table 202 would be used by a human to understand the meaning of theprofile 200; it is not needed for operation of the social robot. Inparticular, the labels in columns 218 and 222 are merely for humaninterpretation, while the labels in columns 216 and 220 merely refer tothe corresponding indices in columns 210 and 214. The negative index incolumns 214 is merely for illustrative purposes, as a reminder that thenumeric personality scores in column 212 pertain to a measurementbetween poles of a personality trait. In practice, only a single indexas shown in column 210 is needed, as its opposite in column 214 may besupplied by implication.

The proportion and intensity with which any particular personality traitor its opposite is expressed by a character is typically neither fixedor always randomly varying. Instead, the expression of any trait more orless (depending on the particular trait) can usually be correlated bothto context in which an event occurs, and the nature of the eventexperienced by the character, among other stimuli. People naturally seekto understand the expression of traits based on context, which mayinclude both objective and subjective factors, and events, which aregenerally objective. For example, a character may be typicallyinterested by airplanes and bored by dolls as a young boy, due to asubjective desire to conform to social norms for the character'ssubjectively-assumed identity. The same character, however, mighttemporarily take an interest in dolls for any number of objectivereasons, for example to please a favorite playmate who wants to play, orbecause a doll has particular features that provoke the character'sinterest. Accurate emulation of a character's personality thereforerequires more than merely expressing particular trait in somequantitative proportion (e.g., the character is empathetic 50% of thetime) no matter the context or events experienced. Instead, accurateemulation may require that the social robot express a particular trait(e.g., empathy) in certain combinations of stimuli, and express anopposite of the trait (e.g., lack of empathy) in other certaincombinations. The technical methods described herein include flatteningmulti-dimensional combinations of stimuli, for example by assigning eachcombination with a number derived by walking the nodes of a directedacyclic graph in which each node represents one factor of the combinedstimuli, as described in connection with FIGS. 7A-B, 8 and 9 below.

Referring again to FIG. 1A, a character analysis module 122 may producetwo outcomes from an analysis process: a list of characteristic socialbehaviors 130, and a set of one or more personality profiles 128, eachcorrelated to a distinct mood of the character. As before, behaviors arestimulus-response pairs. Mood can be regarding as a type of context inwhich stimulus-response events occur. The behavior list 130 is a form ofthe behavioral data 124, processed by analysis 122 to be in the form ofone behavior per record, of every social behavior that is recorded forthe source character. The personality profiles 128 are data structuresas described in connection with FIG. 2 , each assigned to a distinctmood of the analyzed personality. A “mood” may be thought of as a“personality mode”; in technical terms each mood is a distinct set ofpersonality trait values that can be observed for continuous periods andto emerge and/or dissipate with at least some predictability in responseto one or more mood triggers. A character analyses process 122 maydiscover a character's moods by grouping stimulus-response data overdifferent continuous time periods, matching similar behavior overdifferent periods, and identifying triggers for time periods in which adistinct mood is observed. Moods can be used for social robot controlinasmuch as convenient but are not required; the effect of a mood can beaccomplished in other ways, such as, for example, by specifying morespecific stimuli for certain responses.

Personality modes can also change in phases, such as when a characterages. Phases of development are non-cyclical and have much longerperiods than moods. For example, a person when born has an infant'spersonality; later a toddler's, then a preschooler's, and so forth.These phasic personality modes can gradually transition from one to thenext, and each typically occurs only once during a character's life.Some phasic personality modes are age driven and can be modeled based ona character's age. Other phasic personality modes may be fairly abruptand are triggered by significant events, for example, forming or losinga significant social relationship, experiencing social or physicaltrauma, becoming addicted to a psychoactive substance, joining orleaving a social group, achieving or losing a high social status, and soforth. A character's phasic personality mode shifts may be implementedsimilarly to mood shifts as described herein. Unlike moods, however,phasic personality modes are experienced only over periods longer than aday, usually requiring months or years except in rare cases such as whentriggered by a traumatic or other transformative event. Therefore,phasic personality shifts are an option for social robots intended toform varying long-term relationships with their users during which thesocial robot undergoes lifelike phasic personality shifts but may beignored when configuring social robots for other purposes.

FIG. 3 shows an example of a process 300 for character analysis 122. Theprocess 300 provides an example of an algorithm that may be programmedand compiled and supplied on a memory for execution by a computerprocessor, to perform the process 300. Initially, a set of filters 304,308, 312 are applied to the available stimulus-response pairs. Althoughdescribed as sequential operations, it should be appreciated that thefiltering 304, 308, 312 may be performed in parallel as part of the pairidentification 302, or after it. The process 300 may include, at 302,identifying a stimulus-response pair in a record of character behavior.This may require correlating fuzzy sets of stimuli and responses,because whether or not a specific response is in reaction to aparticular stimulus is inherently subjective and known (consciously ornot) only to the character that performs the response. The process 302,however, is objectively performed on fixed data, such as, for example,studio assets such as video clips, scripts, etc. Therefore, identifyingstimulus-response pairs depends on objective criteria, such as aobserved correlation between events, or between states and events. Deeplearning excels at objectively identifying correlations, particularlywhen provided with large data sets, and may therefore be useful for theprocess 302 as applied to voluminous recorded character data. Dialog isa species of stimulus-response that is particularly important to socialinteraction. Linguistic analysis may be used to not only determinesemantic value of dialog, but also non-verbal (or implicit) ranges ofmeaning, for implicit communication such as innuendo, sarcasm, humor,and irony. Linguistic analyses may include, for example, analysis oflinguistic phenomena such as co-occurrence, entrenchment, and emotionaltriggering, as well as semantic and syntactic analysis. Analysis ofintonation, facial expression, body language, and context may also behelpful, together with linguistic analysis, in detecting uses ofimplicit communication. It may be very helpful for personality emulationto be able to detect the use of implicit communication. For example, ifa character uses irony frequently, a faithful emulation of the characterwould require that the social robot make ironical statements at asimilar frequency as the original character, when experiencing similarstimuli.

At 304, the processor may determine whether both the stimulus andresponse are “in range” of the social robot or group of social robotsthat will take on the mapped personality. In this context, “in range”means that that stimulus or response is either found directly in therepertoire of the social robot or can be mapped to a robot-capablestimulus and response. If not in range, the algorithm may, at 306 and302, identify the next stimulus response pair that the data contains. Inan alternative, the algorithm may treat out-of-range pairs nodifferently than in-range pairs, so that filtering of unmappable pairscan be performed in a downstream process (such as the mapping process140).

If in range, the algorithm may determine at 308 whether or not thestimulus and response contain enough detail to fully define a sociallyrelevant behavior. For example, the algorithm may analyze the context ofthe stimulus response and determine whether or not the context revealssocially relevant factors that may affect the response, and whether ornot the response is completely described in adequate detail. If thelevel of detail is too low, the algorithm may increase it at 310, forexample by including more detailed contextual parameters for thestimulus and response in the behavior record.

Context may be handled as part of a stimulus-response pair, or as afilter that limits responses to a subset of possible responses whenapplied. Either way, context is important to both to characterprofiling, where it is needed to obtain an accurate record of behavioralpatterns, and to robot operation, where it is needed to produce accuratebehavioral responses. One aspect of context may include the userprofile. For example, a social robot emulating any particular charactermay interact differently with different types of users, or with users indifferent emotional states. Different user types may include, forexample, the age, gender, and ethnic background of the user. Selectionof real-time social robot response may be such that the interaction ofcertain character profiles with different users produces unique responsepatterns for each different combination of user and character profiles.A social robot that collects user profile data may protect the user'sprivacy by securely encrypting the profile data so that it cannot beused without a user key, and/or some other means of safeguardingpersonal information, for example, an ‘opt-in’ process along withconsumer-friendly usage rules such as secure deletion after some limitedperiod.

If complete, the process 300 may include determining whether thebehavior is characteristic of the behavior under analysis, or at least,that it is not clearly generic or uncharacteristic behavior. This maynot be apparent until sufficient behavior data has been analyzed to makewhat is characteristic appear. Once characteristic patterns appear,these can be used for comparison against behavior that does not seem tobe characteristic, or only very rarely so.

At 314, the processor may determine the present mood. As previouslydescribed, a mood may be recognized in the first place by assessingbehavior data over different time periods and identifying recurring setsof personality traits that appear at different times. The present moodmay be estimated by determining the current dominant personality trait,and more so to the extent congruent with a known mood trigger consistentwith the current mood. In an alternative, or in addition, an emotionaldetection engine (for example Affectiva™, www.affectiva.com) may be usedto detect a source character emotional state, correlating to a mood. At316, the processor may score the stimulus-response pair against apredetermined personality scale. For example, a behavior may be scoredas 75% extroverted, 60% honest, 50% humorous, and so forth. The behaviorpair score and relevant mood may be associated and stored in a datarecord 318.

If, at 320, the current behavior is a mood shift, then the processor mayat 314 record the mood shift parameters in a mood shift record 316. Moodshifts are essentially behaviors like any others, except for beinguniquely well-correlated to changes in personality trait expressionand/or emotional state. At box 318, once all the characteristicbehaviors for the source character have been scored, the processor at320 may combine all of the scores for each behavior and mood andnormalize the resulting sums to the predetermined scale. This operation320 will result in a quantitative personality profile, one for eachmood, which are saved in a record 322. At 324, the processor may collateand combine the behaviors, obtaining the list of characteristicbehaviors 130 that have been previously described, saving them in a datarecord 326. Each behavior in the data record 326 may be associated withits corresponding mood and score that were determined by the precedingprocesses 316 and 316.

Referring back to FIG. 1A, once the source personality has beenanalyzed, the behavior list 130 may be used in the robotic behaviormapping subsystem 140 to prepare a set of personality-correlated roboticsocial behaviors 152. The subsystem 140 may be operated asynchronouslyand partly independently from the profiling subsystem 120; for example,different mapping subsystems may utilize the same source social behaviordata 130 to configure different social robots that have differentcapabilities. Like the profiling subsystem 120, the mapping subsystem140 may be operated manually, semi-automatically, or fullyautomatically, depending on the sophistication of the system 100.

FIG. 4 illustrates aspects of behavioral stimulus-response sets 400 thatmay be helpful for understanding operation of the mapping subsystem 140.The sets 400 may include a set 410 of stimuli that includescharacter-relevant stimuli and robot-only stimuli. The subset 412represents stimuli that the robot is capable of perceiving, while theentire set 410 includes every stimulus under consideration for acharacter and the social robot that will portray it. For example, thestimulus 414 “be in darkness” is not within the robot-capable stimulus412, if the social robot lacks a light detector. In that case, to enablea response to darkness, the character stimulus 414 can be mapped to arobot-capable stimulus 416, to “be told it is dark” for a social robotcapable of speech recognition.

The stimulus set 412 may be paired to a response set 420 of allresponses cataloged for the source personality. Just as with the stimuli410, a subset 422 of the responses are catalogued for the target socialrobot. Some responses are in the repertoire of the source character, butnot possible for the social robot to perform. For example, the “touch”response 424 may not be possible, if the social robot does not have anarticulating limb with a touch sensor. However, the response might bemapped to a response that evokes a similar emotion and is in the socialrobot's capability set 422. For example, the touch response 424 might bemapped to a verbal expression response 426, a “move closer” response(not shown), or to some combination of these or other responses.

Again, referring to FIG. 1A, the robotic behavior mapping subsystem 140may use information 148 about low level routines of the social robot todefine social robot behaviors 144 that are mapped to characteristicsocial behaviors 130 of the source character. For example, the system140 may attempt to construct the closest equivalent robot behavior usinga library of low-level capabilities. The system 140 may also identifyrobotic social behaviors 146 that are generic and are not mapped to thesource character's characteristic behaviors, but that express a socialfunction.

Referring to FIG. 5 , a process 500 for mapping robot behaviors providesan example of an algorithm that may be programmed and compiled andsupplied on a memory for execution by a computer processor, to performthe process 500. At 502, the process 500 may include retrievingstimulus-response pairs for the social robot from a list or similar datastructure in computer memory. At 504, the processor determines whetheror not the robot behavior pair is equivalent to a source pair. In anautomated process, the comparison may include translating theexpressions of the source stimulus-response pairs to the same symboliclanguage as used to express the social robot's stimulus-response pairs.Once expressed in the same symbolic language, the pairs can easily beidentified as congruent, or similar pairs mapped for equivalence. Ifcongruent or equivalent, the processor 506 may define, at 506, acorrelation (e.g., a one-to-one association, a one-to-many association,or a many-to-many association) between a source character behavior andone or more social robot behaviors, or between two or more sourcecharacter behaviors and a social robot behavior or behaviors. In someembodiments, one-to-one associations may be used exclusively. In analternative, if there is no congruence or equivalent to a social robotbehavior, the processor may, at 508, mark or provide another indicationthat there is no mapping between a social robot behavior as acharacteristic source behavior. At 510, the processor may record thesocial robot mapped and unmapped behaviors in a correlations datastructure 512. At 514 and 516, the processor cycles through the nextlisted behavior until all robot behaviors are mapped or indicated asunmapped.

Referring again to FIG. 1A, at 150, the robotic behavior mappingsubsystem 140 may score the characteristic behaviors 144 based on thesame personality scale 126 used for scoring source behavioral data 128.FIG. 6 illustrates a process 600 for scoring robot behaviors. Theprocess 600 represents an algorithm that may be programmed and compiledand supplied on a memory for execution by a computer processor, toperform the process 600. At 602, the process 500 may include retrievingstimulus-response pairs for the social robot from a list or similar datastructure in computer memory. At 604, the processor determines whetheror not one of the social robot behaviors is mapped to a source characterbehavior. At 606, if the robot behavior is mapped to a source behavior,the processor may link or otherwise associate the robot behavior to thepersonality scale score previously determined (e.g., as per example at316 of FIG. 3 ) for the source character behavior to which the robotbehavior is mapped. At 608, if the robot behavior is unmapped, theprocessor may determine an associated personality score for the robotbehavior, based on the personality scale. At 609, the processor mayrecord the associated score for each behavior in a suitable datastructure. At 614 and 616, the processor may cycle through the nextlisted behavior until all robot behaviors are scored against theapplicable personality scale, and the scores are recorded. Once theprocess 600 or equivalent is completed, the subsystem 140 will haveproduced a data set of robotic social behaviors 152 that are scored on aparticular personality scale.

Before discussion how a social robot may make use of the data set 152 inthe real-time behavior subsystem 160, further details of stimulus,response, and personality scores are considered. FIG. 7A provides anexample of multiple parameters that define a stimulus, and a system 700for organizing the stimulus parameters in logical categories that arehierarchical (as in a directed acyclic graph). A top-level category 702should define the stimulus at the highest useful level of generality,with specificity increasing at lower levels. In embodiments (not shown),each level of the graph may consist of only two choices, one of whichmust be true of logical necessity, such as CA′ and ‘not A.’ Incharacterizing a stimulus either in real time (FIG. 1A at 162) or whenanalyzing source character data (FIG. 1A at 122) a classification engineworks its way from the top level of the graph to the lowest level,choosing the node at each level that is most probable. However, if everylower level node is analyzed to be most probably ‘not A’, the engine mayrevert back to the highest level ‘not A’ node and try the other branch,because a long succession of ‘not A’ choices may be indicative of abranching error. For example, suppose a classification engine encountersa pine tree while using a classification graph in which the highest twonode choices are “person” and “not a person.” Suppose the classificationengine erroneously selects “person.” Suppose the next lower nodes on theperson branch are “male” and “not male.” The engine correctly selects“not male”; the next lower nodes on the “not male” branch are “female”and “not female.” The engine correctly selects “not female” and supposethe next lower nodes are “other gendered” and “not other gendered.” Theengine may be configured to recognize a combination of “not male” “notfemale” and “not other gendered” as impossible for a person, andtherefore revert to the higher node and select “not person.”

In the example illustrated by FIG. 7A, the top-level stimulus is arelative position 702 between the robot or character and something else.The remaining parameters can be organized in any number of sublevels(e.g., in a directed acyclic graph). Here for example, the parametersare organized into four groups: a time (when) group 710, an identity(who/what) group 720, a location (where) group 730, and a manner ofperformance (how) group 740. Each of the groups contains a score betweenbipolar opposites of a sort. For example, the time group 710 may containa time-of-day score 712 and a day-of-week score 714, indicating when thestimulus occurs. Each of these scores may be normalized to a numericscore that runs continuously between the endpoints of the chosen scale.For example, if an eight-bit scale is used for a time-of-day parameter712, each 24-hour period from just past midnight until midnight of thefollowing day may be divided into 128 equal incremented, enablingtimekeeping to a granularity of 128/24 (5⅓ units per hour, or 11.25minutes. This should be more than enough granularity for correlatingsocial responses. Using the same eight-bit scale to denote time overdays of the week results in 7 times less granularity, or 78.75 minutes,while enabling mood or other response correlations to days of the week.For further examples, an identity group 720 may include values measuringcharacteristics of the person or thing that is the source of thestimulus (e.g., person who is addressing the robot, household pet, orobject in the center of the robot's field of view), for example, aperson/thing scale 720, a gender scale 724, and an extroversion scale726.

A location group 730 relates where the stimulus occurs, for example anenclosure parameter 732 and a neighborhood character parameter 734. Forfurther example, a manner of performance parameter group 740 may includea proximity measure 742, one or more orientation measures 744 and arelative motion measure 746.

If the identity of the primary user is known or can be estimated, theuser's age 727 may be an important factor in determining the responseselected. The social robot may behave according to a specific fixed age,while modifying its response to be age appropriate for the person orpeople who will witness the robot's response. In addition, or in analternative, the robot's personality, and thus, its responses, may bemodified according to the age of its primary user or owner. For example,a social robot given as a gift to a 5-year-old who is thereafter itsprimary user may operate using a kindergarten personality profile (setof personality traits) for the source character, no matter who it isinteracting with. As the primary user ages, robot's the personalityprofile may be modified according to the source character at an olderage, aging at a rate that generally correlates to aging of its primaryuser. Modification of the robot's personality to reflect aging may bedone continuously, but for computational simplicity, it may bepreferable for aging to proceed in phases. Either way, the social robotcan age with its primary user, becoming a life-long companion with apersonality that matures and evolved based on its source. In analternative, the social robot's personality may be aged but at a ratethat does not correlate to aging of its primary user. For example, therobot's personality may grow older more rapidly or more slowly than itsprimary user, or age in reverse (grow younger). The capability of therobot to adopt a personality that evolves in a way that people canrelate to, whether or not the rate of aging exactly parallels that ofthe primary user. The robot's age or time of operation are furtherexamples of time factors that may be used in determination of therobot's social response.

The robot's experience level 729 with any particular user may be anotherimportant factor in response selection. For example, the robot mayselect a different response depending in part on whether the personinteracted with is a stranger, a new acquaintance, a casualacquaintance, a more intimate friend, or an old friend for whom therobot possesses a long history of experience. In addition, or in thealternative, the robot's specific experience with the user may influencethe selected response. Various robot responses may be designed thatinclude generic or default options, which can be supplemented with morespecific information from the robot's memory if known. For example, ifthe robot has been introduced to the user and knows the user by name,the robot may address the user by name instead of using a generic formof address. The response may be structured using conditional logic suchto select more specific responses if information is available. A morecomplex example is provided by suggesting an activity, or commenting ona past activity, based on conditional selection of information about auser's past activities or interactions with the robot. For example, ifthe robot perceives that the user is bored, it may make a specificsuggestion based on prior activities the user has performed in similarcircumstances or make a generic suggestion if it lacks this pastexperience.

The system 700 is simplified for illustrative purposes, and the specificmeasures included in the parameter examples are less important than theprinciples of organization and quantification that have been described.Under the illustrated principles, any given stimulus can be representedas a matrix of measurement values. For example, each group can provide arow of the stimulus matrix and each column may delineate betweenadjacent parameters, or vice versa; the matrix may have any usefulnumber of dimensions. A matrix with one or two dimensions is believeduseful; higher dimensions increase complexity but may also be useful.Each cell of the matrix may hold an eight-bit (or other) numeric value,the meaning of which depends on the system 700. It should be appreciatedthat using a relatively standard system for different stimuli wouldenable rapid machine comparisons and correlations between differentstimuli and their associated responses. The meaning associated with eachcell is not critical, but the value of the cell should have a materialimpact on the response in at least some cases, or else the parameter isnot worth tracking. Accordingly, a general stimulus organization system700 may be organized to enable use of a standard matrix for all stimuliof interest, while maximizing the impact of each cell value on theassociated social response.

FIG. 7B shows the same organizing and quantification principles appliedto system 750 for characterization of a social response. In thisexample, the top-level category 752 pertains to speech. The time group760 containing one or more “time factor” (also called a time parameter)does not pertain to time of day, etc., because that aspect is controlledby when the stimulus occurs. The main time parameter 762 pertains to howquickly the speaking response is made after the stimulus is received.The identity group 770 may contain numeric measures of the speechcontent, such as the length of speech 772, its humorous character 774,its truth 776, and so forth. The actual words spoken cannot readily berepresented numerically in the organization scheme illustrated by FIGS.7A-B. Instead, for a character that uses distinctive stock phrases, eachstock phrase can be scored based on the combination of content identityparameters in the group 720, and thereby identified by score comparisonwhen socially appropriate for the personality being exhibited. Theposition group 780 may contain parameters 782, 784 that relate thedirection to which the speech is addressed relative to the person orthing being addressed. A manner of performance parameter group 790 mayinclude a pacing parameter 790, a volume parameter 794, and a subtletyparameter 796. Similar organizational principles may be used forstimulus systems (e.g., 700) and response systems (750), but in bothcases care should be taken that the measurement parameter for a stimulusdoes not more truly relate to the response, and vice-versa.

FIG. 8 illustrates aspects of a flat data structure 800 for holdingquantified multi-parameter data for stimulus or a response. A header 801may include an identifier 802 for a top-level stimulus or responsecategory. The header 801 may further include an indication 806 of thetotal number (‘N’) 808 of subclasses used in the data structure 800.Each subclass identifier 810, 814, 818 may correlate to a parametergroup as described above. The header 801 may be followed by a list ofvalues 812, 816, 820 that may each define several distinct parametersthat may be distinguished by bit position. Parameters within the groupmay be specified by a predetermined number of bits in the numeric values812, 816, 820 in a predetermined order. Essentially, the data structure800 comprises a string of digital values with enough index informationto make use of the string.

FIG. 9 illustrates graphically a system 900 in which a 4×6 numericstimulus value 902 is paired to a 4×4 numeric response value 904. Inaddition, the illustrated stimulus-response pair 902, 904 (collectively,a behavior) correlates to a single 1×8 numeric personality score 906 forthe behavior, based on any used scale. In the illustrated example, blackindicates a maximum value, white represents a minimum value, grayrepresents some intermediate value, and “X” represents a null(inapplicable) value. Each cell or box represents a parameter value fora stimulus or response. Hence, the system 900 illustrates how apersonality score 906 can be correlated to one or more numericallyrepresented behaviors, using purely numerical methods.

For example, given a large set of many stimulus-response pairs allnumerically represented under a consistent scheme, a processor maydetermine a correlation coefficient between each cell of the stimulusmatrix and each cell of the response matrix. For each cell of thestimulus matrix, the processor may determine the most positivelycorrelated response parameter and the most negatively correlatedresponse parameter, over the entire behavior set, using a measure suchas, for example, a Pearson's correlation coefficient. For example, thevalue of the stimulus parameter at row-column (1, 1) may correlate mostpositively with the response parameter at (1, 4) and most negativelywith the response parameter at (2, 2). If so, the first two values ofthe personality trait matrix for the behavior set as a whole mayindicate the value of the correlation. For example, the amount ofpositive correlation (e.g., a number between 0 and 1) may be indicatedby seven binary digits plus one digit to indicate the sign (positive ornegative), while the amount of negative correlation can be indicatedusing the same scheme. If this is repeated for every stimulus value, anumeric personality score for the entire behavior set can be determinedwithout any need for a human to assess stimulus-response pairsqualitatively. This may be done for every distinct set of behaviorsidentified with a particular mood, to provide a personality profile foreach mood. In an alternative, a human may score the personality traitvalues based on predefined scales as previously described herein, orsome other numeric method is used that is responsive to correlationsbetween stimuli and responses. The meaning of a purely numericallydefined trait scale will, unlike a predetermined standardized scale,depend entirely on the set of behaviors from which it is derived, andthe numeric method used to derive it. However, this may not be adisadvantage so long as the social robot is programmed to use whateverpersonality profile and scale that are derived and defined for aparticular behavior set, and the numeric method reflects a rationalmeasure of correlation between stimuli and responses for the behaviorset.

A quantitative personality profile that is defined using a purelyautomated, numeric process may not have any meaning that can easily beexpressed in words, unlike the predetermined personality profiles thatare described herein above. The automatically determined profile maynonetheless enable a social robot to behave in a manner that resemblesthe behavior set from which the numeric personality profile is derived,because it will capture the most important correlations between stimuliand responses. Consider, for example, how a behavior (e.g., astimulus-response pair 902, 904) can be scored against a numericallydetermined personality profile. Suppose the described numerical methodis used to derive a personality profile for a set of behaviors. To scorea specific behavior (e.g., pair 902, 904), a processor may compute adifference between each actual response value and a value predicted bythe stimulus value and the correlation coefficient for each parameter ofthe stimulus (e.g., parameter 1,1) based on each of its correspondingmost highly correlated response parameters over the entire behavior set(e.g., the response parameters at 1,4 and 2,2). Repeating this for eachstimulus value, a score can be obtained that indicates how well thespecific behavior conforms to the personality profile that has beennumerically determined for the set. The applicable library of behaviorsassociated with the personality profile may be defined as those withinsome numeric measure of the derived personality profile, as those thatbelong to the set of behaviors from which the personality profile wasderived or based on some combination of the foregoing. Advantageously,use of a numeric measure to determine the range of a personality profileenables the addition of new behaviors to the social robot's repertoire,fitted appropriately to one or more sufficiently matching personalityprofiles.

In general, whether or not a personality profile is derived purelynumerically or is based on some manual scale that ishuman-comprehensible, scoring any particular stimulus-response pairbased on the applicable scale should enable a social robot to lateridentify a proper subset of all of the robot's social behaviorscontaining only those behaviors within a desired degree of similarity(e.g., plus or minus 10%) of a specific personality profile for thecurrent character and mood, or that are flagged as being part of thebehavior set from which the current personality profile was developed.Regardless of how the personality profile is developed, operation of thesocial robot in real time is always fully automated based on algorithmsprogrammed into the robot's control processor(s).

High level modules of social robot operation are indicated in subsystem160 of FIG. 1A. Environmental data 164 represents sensor data for a realsocial robot, and virtual environment data modeled in computer memory,for a virtual social robot. The social robot reacts to its environment,but before calling up an appropriate response based on its personalityand current mood, the robot control processor needs to identify acurrent stimulus set that can be related to stimuli specified in therobot's library of personality scored social behaviors 152. The process152 for stimulus characterization may be used in real time to identifycurrent stimuli from environmental data 164. An example of a process1000 for stimulus characterization is diagrammed in FIG. 10 , forperformance by a processor of a social robot.

At block 1002, the processor receives sensor data (if a real socialrobot) or modeled environmental data (if a virtual social robot) via adata stream or file and records the received data in a cache memory1004. At 1006, the processor analyzes data from the cache and detectsone or more events, using any suitable event detection method as knownin the robotic arts. The processor places identifiers for the detectedevents in an events cache 1008. In parallel to the event detection 1006,the processor compares 1010 contents of the current event cache 1008 tothe robot's stimuli library 1012 in a computer memory, which containsall stimuli associated with any defined social response for the robot.If no stimulus is detected 1016, the process loops back to block 1002until operation of the social robot is finished at 1020. If a stimulusis detected at 1016, the processor signals an alert 1014 that indicatesan identifier for the detected stimulus (e.g., current stimulus 166shown in FIG. 1A) for use by a mood determination and/or responseselection module of the social robot. Once the alert is provided at1014, the processor flushes 1018 the caches 1004 and 1008, and theprocess 1000 loops back to block 1002 until operation of the socialrobot is finished.

Referring again to FIG. 1A, a mood determination process 168 may be usedto switch the current personality profile of the social robot, based onthe current stimulus 166, a subset of mood switching behaviors 153derived from the robot's library of social behaviors 152, and thelibrary of personality profiles correlated to specific moods 128. Itshould be apparent that if the current stimulus does not match any ofthe stimuli included in the library of mood switching behaviors 153 andprovided that an initial or default mood of the social robot haspreviously been determined, the mood determination process 168 may bebypassed until a match is detected.

FIG. 11 illustrates an example of a more detailed process 1100 for mooddetermination, such as may be used in a social robot subsystem 160. At1101, when the social robot is initialized (e.g., when booting up) theprocessor may set an initial or default mood for the social robot,corresponding to a default personality profile, and provide an alert ofthe current mood and profile to a downstream response determinationmodule. At 1102, the processor waits for a stimulus alert as discussedin connection with FIG. 10 . Once an alert is received 1104, theprocessor compares 1106 the identified stimulus to the library ofstimuli specified by a data store of mood switching behaviors 1108. If asufficiently close match is detected 1110, the processor updates thecurrent mood 1112 based on the closest matched stimulus and itsassociated mood. At 1114, the processor selects a personality profilebased on the current mood, using a data store 1118 of personalityprofiles and associated mood identifiers for the social robot. At 1116,the processor provides an alert indicating that the personality profilehas been updated to a specified value. This may include, for example,loading a values or values that define the current personality profileinto a memory location (e.g., a location pointed to by a variable) thatis used by a downstream response determination module. The process 1100then loops back to the wait block 1102 until the social robot operationis finished at 1120, such as, for example, when the social robot ispowered down or placed into an inactive state by the operator orautomatically.

Referring for the final time to FIG. 1A, the current personality profile170 may be provided to a response selection module 172, together withthe current stimulus 166 provided it is not a stimulus formood-switching and nothing else. The response selection process performsa behavior filtering process based on the social robot's library ofsocial behaviors 152 and on other factors used to eliminate behaviorsthat may be inappropriate for the current stimulus (for example,behaviors with a high risk of damaging the robot or harming bystanders).A selected response 174 is indicated to a driver 176 for lower-levelrobotic functions, which executes the response 174 in the real orvirtual robot.

FIG. 12 indicates an example of a more detailed process 1200 forresponse selection, such as may be useful in the social robot subsystem1100. The process 1200 may include, at 1202, a processor of the socialrobot obtaining a current value of the personality profile from a mooddetermination module or process. In alternative embodiments, the socialrobot may obtain multiple personality profiles to be blended or used insequence, a or single profile that has been generated by blending two ormore distinctive profiles. At 1204, the processor may wait for astimulus alert from a stimulus characterization module or process suchas, for example, previously described. Once a stimulus is alerted at1206, the processor may select responses from the library of socialbehaviors 1208 that most closely correspond to the current stimulus andpersonality profile. As previously described, a stimulus may be in theform of digital data representing a number series or matrix, as may alsobe a personality profile. If more than one personality is being used tocontrol the response, the two or more profiles may be blended byaggregating (e.g., averaging or interpolating) values of differentprofiles, or by selecting one profile for some stimuli, and anotherprofile for other stimuli. A simpler blending process may use physicaltraits (e.g., voice, appearance) from one profile, and behavior fromanother. These and other blending approaches may also be used incombination.

The selection process 1210 may include a filtering operation based onidentifying the stimuli from the library 1208 that match the currentstimulus value, either exactly or within a specified range, and anotherindependent filtering operation that identifies the stimuli that arewithin a specified range of the current personality profile value, orthat are flagged as belonging to the current personality profile. Theprocessor may select only those stimuli that pass both filteringoperations. In addition, the processor may apply other filteringcriteria, such as, for example, criteria based on preserving robot oroperator safety.

If only one stimulus matches 1212, the processor identifies the responsethat is paired with that stimulus and alerts it 1216 for downstreamexecution by the social robot. If more than one match is detected, theprocessor selects one of the matched responses 1214 based on anassociated matched stimulus value, for example, by using random orquasi-random selection, or any other desired selection method. Inembodiments, the processor may rank the possible response, and selectthe highest-ranking response. For example, the processor may calculate aprobability score for each of the possible social responses, based on aset of correlation factors stored in the memory. Each of the correlationfactors scores one of the possible high-level responses relative to apersonality trait scale used for the scaling the set of quantitativepersonality trait values. The process 1200 then loops back to the waitblock 1204 until the social robot operation is finished at 1218, suchas, for example, when the social robot is powered down or placed into aninactive state by the operator or automatically.

The subsystem 160, including but not limited to the stimuluscharacterization 162, mood determination 168, and response selection172, may be implemented as modules in, or processes by, a real socialrobot 1300 for example as shown schematically in FIG. 13 . The socialrobot 1300 may include electro-mechanical components including anelectrical circuit configured as a computer housed in an exoskeleton orhousing 1330. The circuit may include a central processing unit or units1302, operatively coupled to motor driver controller 1320 driving amotor 1322, which moves load comprising a mechanism connected to a heador body portion of the social robot 1300, causing the head or bodyportion to move in a socially significant way. The processor determinesresponses and provides motion control signals to the motor driver 1320,based on a behavioral response selected as described herein above. Themotion control signals cause the motor 1322 to move the connected bodypart in conformance with the response selected by the processor 1302.

The robot 1300 may contain other components for computing in addition tothe processor 1302. For example, the processor circuit may be coupled toa user interface component 1304, such as for example a keypad ortouchscreen, for entry of user controls. The processor circuit may becoupled to a display component 1310, for example a touchscreen or otherdisplay. The display 1310 may be used for a user interface, to displayexpressive features of the social robot (e.g., a face), or for both. Theprocessor circuit may be coupled to one or more sensors 1306 configuredfor sensing one or more environmental conditions of the environmentaround the robots, or of objects that the robot interacts with. The oneor more sensors 1306 may include, for example, a camera, light sensor,microphone, temperature sensor, pressure sensor, accelerometer, touchsensor, proximity sensor, or other sensing device. The processor circuitmay be coupled to a communications port 1308 four coupling to a computernetwork, or to a data communication network for receiving data andprogram instructions. The processor circuit may further be coupled to anaudio output transducer 1318 for outputting robot vocalizations. Theprocessor circuit may further be coupled to a volatile random-accessmemory device 1314 and to a non-volatile memory 1316 for long termstorage of program instructions and data.

An example of a real social robot 1400 is shown in isometric view inFIG. 14 . The social robot 1400 comprises an assembly of a smartphone1402 (or similar computing and wireless communication device) with amechanical base. The smartphone 1402 couples via a wired or wirelessinterface to a power and control circuit in the base 1404. Separately ortogether, the smartphone 1402 and base unit 1404 include an electronicsubstrate holding one or more processors coupled to a memory and to atleast one circuit for driving components of the robot. A circuit in thebase unit may be similar to the circuit described in connection withFIG. 13 , minus a touchscreen display which is supplied by thesmartphone. The circuit in the smartphone may also be like the circuitof FIG. 13 , minus the motor and motion components. Together, the baseunit and smartphone comprise a circuit with all of the componentsdepicted in FIG. 13 . For example, the robot's sensors may include acamera 1426 mounted in the body of the smartphone 1402.

In some embodiments, the main processor and application for controllingoperation of the social robot is located in the smartphone 1402, with aprocessor in the base unit 1404 operating as a slave or client system.In alternative embodiments, the main processor and application forcontrolling operation of the social robot is located in the base unit1404, with the smartphone 1402 playing a client or slave role. The dualdevice, modular construction provides an advantage of enabling acommonly available, versatile device (the smartphone) to supply manynecessary components of the robot 1400, while still allowing it to beused as a smartphone at other times. Nonetheless, the methods herein forcontrolling real social robots apply equally well to modular and unitarydesigns for robots.

In the robot 1400, a face component comprises the display screen 1416,human-recognizable expressive features 1418 controlled by the smartphoneprocessor, including a configurable brow 1420, a configurable pair ofeyes 1422 including at least one moveable eye (e.g., rotatable orsimulated rotatable eye), and a configurable mouth 1424. Thesecomponents may be animated based on a personality profile for acharacter to be emulated. The robot 1400 further includes a movable headcomponent coupled to the face component. The moveable head componentincludes the smartphone 1402. While the robot 1400 lacks a humanoidtorso or appendages, these may be added if desired. The movable head(phone 1402) itself may be controlled at least in part by a processor inthe smartphone 1402, while movement of the base unit 1404 and itsincluded rotating “neck” component 1406 may be controlled at least inpart by a processor in the base unit 1404, that is in communication withthe processor in the smartphone 1402.

The robot 1400 may further include an audio output transducer (notshown) coupled to a control circuit of the smartphone 1402. In thealternative, or in addition, an audio output transducer may be locatedin the base unit 1404. Either or both audio output transducers may beused for the robot's social vocalizations. An audio transducer may alsobe used as a component of a proximity sensing sonar system.

The base housing 1408 may be configured as a torso component coupled tothe head component (smartphone 1402) via a neck mechanism 1406controlled by one or more processors in the base unit 1404. The neckmechanism 1406 permits rotation of the head component 1402 relative tothe torso component 1408. The torso component (housing) 1408 mayconnected to a control circuit via motor and motor drivers locatedinside the base housing 1408. The base housing 1408 may be supported bythree or more wheels (one shown at 1410), at least one of which isdriven by a motor under control of the robot's central processor, and atleast one of which can be steered. Accordingly, the housing 1408 canmove and rotate over any navigable hard surface. The “head” 1402 canalso turn to face any user, person, or object in the area, beingprovided with two degrees of rotational freedom by the neck and torsocomponents 1406, 1408.

To prevent collisions or falls, a pair of proximity sensors 1412, 1414and/or the camera 1426 may provide navigation data to a navigationalengine executing in a processor of the robot. The navigational enginemay limit or alter movement of the movable torso 1408 and neck 1406 toprevent collisions and falls, while otherwise providing output motion inconformance with requested social behaviors. The combination of bodilymovement, neck rotation, facial expression and audio output enables therobot 1400 to express a surprisingly complex personality, which mayreadily be recognized as derived from a fictional character orperformance.

To that end, a computer memory of the robot 1400 may hold programinstructions, that when executed by its one or more processors, causesthe social robot to control its social responses to environmentalstimuli based in part on a set of quantitative personality trait valuesstored in the memory. The quantitative personality profile andassociated library of robot behaviors may be developed and configured asdescribed herein above. Accordingly, the social robot 1400 may be causedto behave in a manner consistent with the character from which thepersonality trait values are derived, when interacting socially (e.g.,with a live companion).

The personality emulation techniques described herein may be appliedequally well to virtual social robots, which can be caused to simulatein their virtual environment many real behaviors and interactions. Inaddition, virtual social robots may exhibit extended behaviors (e.g.,flying, shrinking, expanding) in manners that are not feasible for realcharacters or for real social robots. These extended behaviors can bemapped to real character behaviors as described herein above. Hence, avirtual social robot can be caused to extend the behavior of a realcharacter, in ways that conform recognizably to the real character'spersonality. In addition, a virtual social robot can readily performmost or all behaviors of entirely fictional characters, such as talkingcats or superheroes endowed with supernatural or exaggerated powers. Thestimulus used to provoke virtual robot behavior may include the actionsof a user-operated avatar that interacts with the social robot withinits virtual environment, or multiple such avatars, the actions of one ormore non-player characters (e.g., other virtual robots) in the virtualenvironment, the bodily movements or electrical activity of a user orusers wearing or carrying immersive gear for virtual reality oraugmented reality, or any combination of the foregoing stimulus types.

A virtual social robot may include well known components from video gamecharacters or figures in virtual or augmented reality. For example, FIG.15 shows hardware components of a virtual social robot 1500, in blockdiagram form. The virtual social robot 1500 includes a control processor1502, that may be used to generate the virtual environment and thevirtual social robot in it, or that may be dedicated exclusively to thesocial robot or to the social robot and some limited part of its virtualenvironment. The processor 1502 may be coupled to a short-term memory1504 (e.g., RAM) enabling execution of program instructions, and anon-volatile data store 1506 for long-term storage of virtual data andprogram instructions. The processor 1502 may be coupled to acommunications port for sending and receiving data via a datacommunications or computer network.

The virtual social robot 1500 may be expressed via at least two outputs:audio and video. The social robot 1500 may include an audio drivercomponent 1512 coupled to the processor 1502, producing an audio signal1515 for providing to an audio output transducer 1520. The social robot1500 may also include graphics driver component 1510 coupled to theprocessor 1502, producing a video signal 1515 for providing to a videodisplay device 1518.

The virtual social robot 1500 may hold program instructions in thememory 1504, that when executed by the processor 1502, causes the socialrobot to respond to virtual environmental stimuli based in part on a setof quantitative personality trait values stored in the memory 1504and/or data store 1506. The quantitative personality profile andassociated library of robot behaviors may be developed and configured asdescribed herein above. Accordingly, the social robot 1500 may be causedto behave in a manner consistent with the character from which thepersonality trait values are derived, when interacting socially (e.g.,with a live companion).

FIG. 16 shows an example of hardware or firmware components 1600 of avirtual social robot. Hardware components of the robot may be consistentwith those shown and described in connection with FIG. 15 . The softwareof firmware components 1600 may include a geometric mesh 1602, anarmature or system of bones 1604, a set of mesh deformation rules 1606governing how the mesh responds to movement of the armature 1604, andsurface rendering data 1608 (e.g., diffuse, specular and bump orelevation maps). The software or firmware components may include anysuitable elements as known in the art of model manipulation andrendering. The components may be configured in any desired way toresult, when rendered, in a virtual social robot of any desiredappearance 1610. A single virtual robot having a certain personality mayeven take on different appearances as a form of social expression.Technical aspects of manipulating and rendering virtual characters arewell understood in the art and may readily be adapted to make use ofpersonality modeling and expression techniques as described herein forsocial robots generally.

Referring to FIG. 17 , a real social robot and a virtual social robotmay be coordinated 1700 to represent a unitary real-virtual socialrobot. For example, a real mobile social robot carrying a locatingdevice might move around a physical play space for a virtual realitygame (e.g., by moving over the floor, flying through the air, climbingwalls, or combinations of these or other motions). One or more userseach wearing virtual reality (or augmented reality) gear may see andhear a virtual social robot rendered in a virtual environment (oraugmenting a real environment) at a location corresponding to a currentlocation of the real social robot. The real social robot may move inresponse to the user actions in the real environment and in the virtualenvironment. For example, a game of Quidditch might be played in thisway, or a similar game in which the Quidditch snitch is replaced by acharacter with personality, such as, for further example, a game of“capture Bugs Bunny.” Many other use cases for mixed social robots arealso possible. In each of these use cases, a control processor for themixed reality session may allocate a unitary (total) behavior responseset 1702 to a real response set 1704 and to a virtual social robotsresponse set 1706. For example, overall motion response may be allocatedto the real robot, and other responses (e.g., facial expressions andspeech) allocated to the virtual robot. Stimuli is for triggering theresponse may, conversely, be detected by aggregating stimuli detected inthe real and virtual contexts. The control processor may furthersynchronize the response sets 1704, 1706 so that the responses of thereal social robot and the virtual social robot are coordinated. In otherrespects, the preparation of the total behavior sets (includingresponses), personality profiling for source character, and behaviorselection may be performed as herein above.

In view the foregoing, and by way of additional example, FIG. 18 showaspects of a method 800 or methods for control of a social robot basedon a prior character portrayal in fiction or performance, based on aquantified personality profile. The method 1800 may be performed by acomputer processor controlling a real social robot, a mixed realityoutput device, or combination of the foregoing, and components connectedto the processor as described herein above. The method 1800 may include,at 1810, recognizing, by a processor coupled to a source of informationabout events experienced by a robot, a stimulus belonging to a set ofpredefined stimuli, based on a set of events experienced by the robot.Recognizing may include more detailed operations, for example asdescribed above in connection with FIG. 10 , or in the paragraphs below.

The method 1800 may further include, at 1820, selecting a response tothe stimulus, based at least in part on a current personality profile ina computer memory operatively coupled to the robot, wherein thepersonality profile comprises a set of quantitative personality traitvalues and the response is a social response, as described in moredetail herein. The selecting 1820 may include response selection asdescribed, for example, in connection with FIG. 12 . The method 1800 mayfurther include, at 1830, causing the robot to perform the response. Forexample, the processor may send a set of commands to lower level devicedrivers or modules, to which the drivers or modules respond byperforming the response actions.

In an embodiment of the method, the robot includes an electronicsubstrate holding one or more processors coupled to a memory and to atleast one circuit for driving components of the robot. In suchembodiments causing the robot to perform the response may includesending an electrical signal to the circuit, wherein the signal isconfigured to cause a motor connected to the circuit to move a part ofthe robot (e.g., a head, neck, or appendage). In addition, or in analternative, the robot includes an electronic substrate holding theprocessor operatively coupled to a memory and to a face component, theface component comprising at least two human-recognizable expressivefeatures controlled by the processor and selected from: a configurablebrow, at least one configurable eye, and a configurable mouth. In suchembodiments, causing the robot to perform the response may includesending an electrical signal to the face component, the signalconfigured to cause movement of the expressive features. For example,the face component may include a video display screen, and the sendingthe electrical signal comprises sending a video signal that encodes atleast two human-recognizable expressive features and causing a displaydevice to display those features in a facial expression. In an aspect,the source of information comprises one or more physical sensors coupledto the processor, and the operation of recognizing 1810 furthercomprises determining the set of events based on data from the one ormore sensors. Further aspects of real social robots may be as describedherein above.

In an alternative embodiment, the robot includes a processor coupled toa memory, to a user input port, and to a video output device, forexample, a video screen, projector, virtual reality headset, oraugmented reality headset. The user experience of the social behavior ofthe robot may be entirely virtual. In such embodiments causing the robotto perform the response may include modeling an animation of a model ina virtual computer environment, rendering the environment and robot, andsending an electrical signal to the video output device, causing it tooutput a video or equivalent moving image of the robot performing theresponse. In such embodiments, the source of information about eventsexperienced by the robot may include a computer model of the virtualenvironment in which the robot acts and recognizing the stimulus mayinclude tracking activity of the computer model.

In another aspect of the method 1800, the recognizing may furtherinclude comparing contents of an event cache containing data thatdefines the set of events experience by the robot to a stimuli librarythat may be stored in a long-term computer memory and that defines allstimuli associated with any defined social response of the robot. Therecognizing may further include detecting a match between the contentsof the event cache and a stimulus from the stimuli library. After amatch is detected, the event cache may be cleared.

In another aspect of the method 1800, the personality profile mayinclude an aggregate of scores each derived by scoring a set ofstimulus-response pairs observed for a source character, based on ascale of personality traits. The aggregate may include any suitablenumerical aggregate measure, for example, an average or median, based ona rational combination of scores.

In another aspect of the method 1800, selecting the response may furtherinclude selecting candidate responses (meaning possible responses) basedon predetermined associations between each of the candidate responsesand the stimulus. Each stimulus may be associated multiple responses,thus permitting a personality of the robot to vary depending on which ofthe multiple associated responses is determined by the robot processorto consistent with the current personality profile and avoidingpredetermined responses to any given stimulus.

In another aspect of the method 1800, selecting the response may furtherinclude determining behavior scores for each of the candidate responses,wherein each of the behavior scores relates to a different and distinctstimulus-response pair selected from a set consisting of the stimulusand the candidate responses. Each of the behavior scores may also bebased on the scale of personality traits; the same units and mode ofmeasurement may be applied to behaviors as to personalities, with thedistinction that a personality is an aggregate measure that can be basedon multiple behavior scores. Selecting the response may further includecomparing each of the behavior scores to the personality profile, andselecting the response based on the comparing. For example, selectingthe response may further include selecting the response based on whichof the behavior scores is numerically closest to the personalityprofile, using any suitable distance or similarity measure such as, forexample, Euclidean distance, squared Euclidean distance, Manhattandistance or Minkowski distance.

In another aspect, selecting the response may further include selectingone of alternative responses based on a random or quasi-random input.For example selecting the response may further include excluding anyresponse belonging to a stimulus-response pair that is inconsistent withthe personality profile, leaving plural responses that are notinconsistent with the profile. One of these remaining responses may beselected using a random or quasi-random selection, subject to excludingresponses that are not feasible for other reasons (e.g., would createtoo great a risk of harm to the user or to the robot).

In another aspect, selecting the response may include updating thecurrent personality profile of the robot to be a different personalityprofile that is associated with a transient robot mood. For example, themethod 1800 may include a mood determination operation as described inconnection with FIG. 11 , wherein each “mood” is a distinct set ofpersonality trait values that can be observed for continuous periods andemerges and/or dissipates with at least some predictability in responseto one or more mood triggers. Mood selection may be as described in moredetail herein above.

In another aspect of the method 1800, ones of the stimuli may becharacterized at least in part by personal attributes of charactersinteracting with the robots, causing the response of the robot to dependat least sometimes on the personal attributes. The personality profileor mood of the social robot may accordingly be “contoured” to itspresent circumstances. For example, ones of the stimuli may becharacterized at least in part by the type of environment in which anaction included in the stimulus is performed on the robot, causing theresponse of the robot to depend at least sometimes on the environment inwhich the action is performed.

The method 1800 may be adapted or varied. For example, responses may bedivided into classes, such as high level and low level responses. Highlevel responses of a social robot include those responses that aresocially significant to a human companion. A social robot includeshuman-recognizable expressive features controlled by its controlfunction, that express emotion by movement and configuration, forexample a configurable brow, a configurable eye or pair of eyes, amovable head, and a configurable mouth. Examples of high-level responsesinclude orientation of a head relative to torso, orientation of headrelative to human companion, orientation of an eye (gaze direction),configuration of human-recognizable expressive features, speed or waycomponents are moved or configurations are changed, type of verbalresponse, and phrasing of verbal response.

A robot memory may include a table or other data structure thatindicates a correlation between each of the defined high-level responsesand the personality scale used to profile the character that is to beemulated. For example, one high level response may be “pause beforereacting to a stimulus.” The “pause” response may be correlated to thetraits “slow” and “cautious.” For further example, another socialresponse may be “sarcastic reply.” The “sarcastic reply” response may becorrelated to the personality traits “humorous” and “vulgar.” Eachsocial response may be similarly coded in the robot's response selectiondata bank, providing a correlation table or score.

A response selection engine executed by the robot's control processormay make use both high- and low-level filters for selecting robotresponses. Low-level filters may include, for example, excludingresponses that are physically impossible in the robot's currentposition, or that make no logical sense in the context of a conversationthat the robot is engaged in. Conversely, when configured for ironic orhumorous responses, filters may be set to deliberately include illogicalresponses for comedic effect. Once the low-level filter or filters areapplied, a set of possible social responses remain. The robot controlfunction may then assess the probability of each possible socialresponse based on the current personality profile and the correlationtable. For example, the processor may multiply each entry in thecorrelation table by the applicable personality trait factor, sum theresult, compare the sums, and choose the social response with thegreatest sum. For further example, suppose “pause” correlates with“slow” and “cautious,” the robot will compute a high probability of a“pause” response if the personality factors for those traits are high,and a low probability if the personality factors for those traits arelow. Other factors may also be applied, in addition to probability basedon the correlation table and personality profile. Once the most probableresponse is selected, the processor executes the most probable responseusing a set of lower level commands.

FIG. 19 is a conceptual block diagram illustrating components of anapparatus or system 1900 for control of a social robot based on a priorcharacter portrayal in fiction or performance, based on a quantifiedpersonality profile, as described herein. The apparatus or system 1900may include additional or more detailed components for performingfunctions or process operations as described herein. For example, theprocessor 1910 and memory 1916 may contain an instantiation of a processfor mood determination as described herein above. As depicted, theapparatus or system 1900 may include functional blocks that canrepresent functions implemented by a processor, software, or combinationthereof (e.g., firmware).

As illustrated in FIG. 19 , the apparatus or system 1900 may comprise anelectrical component 1902 recognizing one stimulus of a set ofpredefined stimuli, based on a set of events experienced by a socialrobot. The component 1902 may be, or may include, a means for saidrecognizing the stimulus. Said means may include the processor 1910coupled to the memory 1916, the processor executing an algorithm basedon program instructions stored in the memory. Such algorithm may includea sequence of more detailed operations for recognizing a stimulus, forexample, as described in connection with FIG. 10 .

The apparatus 1900 may further include an electrical component 1904 forselecting a response to the stimulus, based at least in part on acurrent personality profile in a computer memory operatively coupled tothe robot, wherein the personality profile comprises a set ofquantitative personality trait values. The component 1904 may be, or mayinclude, a means for said selecting the response. Said means may includethe processor 1910 coupled to the memory 1916 and to at least one sensor(not shown), the processor executing an algorithm based on programinstructions stored in the memory. Such algorithm may include a sequenceof more detailed operations, for example, as described in connectionwith FIG. 12 .

The apparatus 1900 may further include an electrical component 1906 forcausing the robot to perform the selected response. The component 1906may be, or may include, a means for said causing. Said means may includethe processor 1910 operatively coupled to the memory 1916, and to one ormore of the motor driver 1912, the audio adapter 1913, or the displayadapter 1914, the processor executing an algorithm based on programinstructions stored in the memory. Such algorithm may include a sequenceof more detailed operations, for example, sending a set of commands to alower level device drivers or module, and the drivers or modulesexecuting the processor commands by sending signals causing one or moreconnected output devices to perform the response actions.

The apparatus 1900 may optionally include a processor module 1910 havingat least one processor. The processor 1910 may be in operativecommunication with the modules 1902-1906 via a bus 1913 or similarcommunication coupling. In the alternative, one or more of the modulesmay be instantiated as functional modules in a memory of the processor.The processor 1910 may initiate and schedule of the processes orfunctions performed by electrical components 1902-1906.

In related aspects, the apparatus 1900 may include a network interfacemodule (not shown) operable for communicating with system componentsover a computer network, instead of or in addition to the transceiver1912. A network interface module may be, or may include, for example, anEthernet port or serial port (e.g., a Universal Serial Bus (USB) port).In further related aspects, the apparatus 1900 may optionally include amodule for storing information, such as, for example, a non-transitorycomputer readable medium or memory device 1916. The non-transitorycomputer readable medium or the memory module 1916 may be operativelycoupled to the other components of the apparatus 1900 via the bus 1913or the like. The memory module 1916 may be adapted to store computerreadable instructions and data for effecting the processes and behaviorof the modules 1902-1906, and subcomponents thereof, or the processor1910, or the method 1800 and one or more of the additional operations1000, 1100, or 1200 disclosed herein. The memory module 1916 may retaininstructions for executing functions associated with the modules1902-1906. While shown as being external to the memory 1916, it is to beunderstood that the modules 1902-1906 can exist within the memory 1916or an on-chip memory of the processor 1910.

The apparatus 1900 may include a transceiver (not shown) configured as awireless transmitter/receiver, or a wired transmitter/receiver, fortransmitting and receiving a communication signal to/from another systemcomponent. In alternative embodiments, the processor 1910 may includenetworked microprocessors from devices operating over a computernetwork. In addition, the apparatus 1900 may include an audio adapter1913 for providing an audio signal to an audio output device, causingthe output device to vocalize a social robot response that conforms toits personality profile. The apparatus 1900 may include display adapter1914 for providing a video signal for a virtual environment, a socialrobot response that conforms to its personality profile, or both. Theapparatus 1900 may include a motor driver for providing electrical powerto a motor, causing the motor to move a part of the social robot in amanner that conforms to social robot's personality profile.

Those of skill would further appreciate that the various illustrativelogical blocks, modules, circuits, and algorithm steps described inconnection with the aspects disclosed herein may be implemented aselectronic hardware, computer software, or combinations of both. Toclearly illustrate this interchangeability of hardware and software,various illustrative components, blocks, modules, circuits, and stepshave been described above generally in terms of their functionality.Whether such functionality is implemented as hardware or softwaredepends upon the particular application and design constraints imposedon the overall system. Skilled artisans may implement the describedfunctionality in varying ways for each particular application, but suchimplementation decisions should not be interpreted as causing adeparture from the scope of the present disclosure.

As used in this application, the terms “component”, “module”, “system”,and the like are intended to refer to a computer-related entity, eitherhardware, a combination of hardware and software, software, or softwarein execution. For example, a component or a module may be, but are notlimited to being, a process running on a processor, a processor, anobject, an executable, a thread of execution, a program, and/or acomputer. By way of illustration, both an application running on aserver and the server can be a component or a module. One or morecomponents or modules may reside within a process and/or thread ofexecution and a component or module may be localized on one computerand/or distributed between two or more computers.

Various aspects will be presented in terms of systems that may includeseveral components, modules, and the like. It is to be understood andappreciated that the various systems may include additional components,modules, etc. and/or may not include all the components, modules, etc.discussed in connection with the figures. A combination of theseapproaches may also be used. The various aspects disclosed herein can beperformed on electrical devices including devices that utilize touchscreen display technologies, heads-up user interfaces, wearableinterfaces, and/or mouse-and-keyboard type interfaces. Examples of suchdevices include VR output devices (e.g., VR headsets), AR output devices(e.g., AR headsets), computers (desktop and mobile), smart phones,personal digital assistants (PDAs), and other electronic devices bothwired and wireless.

In addition, the various illustrative logical blocks, modules, andcircuits described in connection with the aspects disclosed herein maybe implemented or performed with a general purpose processor, a digitalsignal processor (DSP), an application specific integrated circuit(ASIC), a field programmable gate array (FPGA) or other programmablelogic device, discrete gate or transistor logic, discrete hardwarecomponents, or any combination thereof designed to perform the functionsdescribed herein. A general-purpose processor may be a microprocessor,but in the alternative, the processor may be any conventional processor,controller, microcontroller, or state machine. A processor may also beimplemented as a combination of computing devices, e.g., a combinationof a DSP and a microprocessor, a plurality of microprocessors, one ormore microprocessors in conjunction with a DSP core, or any other suchconfiguration.

Operational aspects disclosed herein may be embodied directly inhardware, in a software module executed by a processor, or in acombination of the two. A software module may reside in RAM memory,flash memory, ROM memory, EPROM memory, EEPROM memory, registers, harddisk, a removable disk, a CD-ROM, digital versatile disk (DVD),Blu-ray™, or any other form of storage medium known in the art. Anexemplary storage medium is coupled to the processor such the processorcan read information from, and write information to, the storage medium.In the alternative, the storage medium may be integral to the processor.The processor and the storage medium may reside in an ASIC. The ASIC mayreside in a client device or server. In the alternative, the processorand the storage medium may reside as discrete components in a clientdevice or server. In another alternative, operational aspects disclosedherein may be embodied in a stream of software that is fed in real timeor near-real time to the robot hardware that is then executed by aprocessor or software module, or a combination of the two. In thismanner, computational power can be off-loaded to the cloud so that therobot's on-board processing can be limited while the cloud-basedcomputational power can be virtually unlimited, allowing moresophisticated inputs, subsequent analyses, recognition and/or responses,and related computational tasks to be accomplished in the cloud.

Furthermore, the one or more versions may be implemented as a method,apparatus, or article of manufacture using standard programming and/orengineering techniques to produce software, firmware, hardware, or anycombination thereof to control a computer to implement the disclosedaspects. Non-transitory computer readable media can include but are notlimited to magnetic storage devices (e.g., hard disk, floppy disk,magnetic strips, or other format), optical disks (e.g., compact disk(CD), DVD, Blu-ray™ or other format), smart cards, and flash memorydevices (e.g., card, stick, or other format). Of course, those skilledin the art will recognize many modifications may be made to thisconfiguration without departing from the scope of the disclosed aspects.

The previous description of the disclosed aspects is provided to enableany person skilled in the art to make or use the present disclosure.Various modifications to these aspects will be readily apparent to thoseskilled in the art, and the generic principles defined herein may beapplied to other embodiments without departing from the spirit or scopeof the disclosure. Thus, the present disclosure is not intended to belimited to the embodiments shown herein but is to be accorded the widestscope consistent with the principles and novel features disclosedherein. In view of the exemplary systems described supra, methodologiesthat may be implemented in accordance with the disclosed subject matterhave been described with reference to several flow diagrams. While forpurposes of simplicity of explanation, the methodologies are shown anddescribed as a series of blocks, it is to be understood and appreciatedthat the claimed subject matter is not limited by the order of theblocks, as some blocks may occur in different orders and/or concurrentlywith other blocks from what is depicted and described herein. Moreover,not all illustrated blocks may be required to implement themethodologies described herein. Additionally, it should be furtherappreciated that the methodologies disclosed herein are capable of beingstored on an article of manufacture to facilitate transporting andtransferring such methodologies to computers.

1. A social robot, comprising: one or more processors coupled to amemory and to at least one circuit for driving components of the robot;wherein the memory holds instructions, that when executed by the one ormore processors, causes the social robot to: control social responses ofthe social robot to environmental stimuli, based in part on a set ofquantitative personality trait values stored in the memory, wherebybehavior of the social robot is consistent with a personality of asource character expressed by a prior character portrayal in a cinematicor video program, and wherein each of the quantitative personality traitvalues represents a score of the prior character portrayal against ascale that measures the each of the personality trait values forcorresponding ones of stimulus-response pairs applicable to the priorcharacter portrayal in the cinematic or video program.
 2. The socialrobot of claim 1, wherein the instructions provide for user-initiatedmodification of the set of quantitative personality trait values,causing the social robot to behave in a manner consistent with a secondcharacter different from the source character.
 3. The social robot ofclaim 1, wherein each member of the set of quantitative personalitytrait values characterizes a personality trait based on a bipolar scalebetween opposite extremes.
 4. The social robot of claim 3, wherein thebipolar scale between opposite extremes, for each member of the set ofquantitative personality trait values, is defined by distinct traitpairs.
 5. The social robot of claim 1, wherein the memory further holdsdata for generating verbal responses based on a set of characteristicphrases for the source character stored in the memory.
 6. The socialrobot of claim 1, wherein the social responses comprise a configurationof a human-recognizable expressive feature, a type of verbal response,and a phrasing of verbal response.
 7. The social robot of claim 1,further comprising a video screen displaying the at least twohuman-recognizable expressive features.
 8. The social robot of claim 1,wherein the instructions are further configured for selecting a socialresponse from a set of possible social responses to an environmentalstimulus stored in the memory, based on the set of quantitativepersonality trait values.
 9. The social robot of claim 8, wherein theinstructions are further configured for calculating a probability scorefor each of the possible social responses, based on a set of correlationfactors stored in the memory, wherein each of the set of correlationfactors scores one of the possible social responses relative to apersonality trait scale used for the scaling the set of quantitativepersonality trait values.
 10. The social robot of claim 9, wherein theinstructions are further configured for the selecting a high levelresponse, at least in part based on the probability score for each ofthe possible social responses. 11.-31. (canceled)
 32. A non-transitorycomputer-readable medium for configuring a social robot based on a priorcharacter portrayal in fiction or performance, comprising instructionsthat when executed by at least one processor cause a social robot toperform: accessing, by the at least one processor, behavioral datacharacterizing a prior character portrayal in a cinematic or videoprogram; deriving, by the at least one processor, quantitativepersonality trait values from the behavioral data, each measuring acorresponding personality trait based on one or more stimulus-responsepairs included in the prior character portrayal; and recording apersonality profile for the social robot comprising the quantitativepersonality trait values mapped to the stimulus-response pairs in amemory for access by the social robot, configured for use by a processorof the social robot for selecting social responses to stimuli thatemulate the prior character portrayal based at least in part on thepersonality trait values.
 33. The non-transitory computer-readablemedium of claim 32, wherein the deriving instructions further compriseinstructions for evaluating the behavioral data against a personalityscale unique to the social robot.
 34. The non-transitorycomputer-readable medium of claim 32, wherein the deriving instructionsfurther comprise instructions for using artificial intelligence torecognize the stimulus-response pairs.
 35. The non-transitorycomputer-readable medium of claim 32, wherein the deriving instructionsfurther comprise instructions for using artificial intelligence tocorrelate the stimulus-response pairs to the quantitative personalitytrait values.
 36. The non-transitory computer-readable medium of claim32, wherein the deriving instructions further comprise instructions forincluding a data object for determining a social response for respondingto combinations of stimuli.
 37. The non-transitory computer-readablemedium of claim 32, wherein the personality profile comprises multiplepersonality modes each correlated to one of distinct behavior patternsto be emulated by the social robot.
 38. The non-transitorycomputer-readable medium of claim 37, wherein each of the distinctbehavior modes comprises a distinct set of personality trait values in amemory coupled to the at least one processor.
 39. The non-transitorycomputer-readable medium of claim 38, further comprising instructionsfor configuring the distinct behavior patterns by grouping thebehavioral data over different continuous time periods of the priorcharacter portrayal.
 40. The non-transitory computer-readable medium ofclaim 39, wherein the instructions further define that the differenttime periods are selected to model moods of the prior characterportrayal.
 41. The non-transitory computer-readable medium of claim 32,wherein the deriving instructions further comprise instructions fordetermining whether ones of the stimulus-response pairs are in range ofa social response profile for the social robot.